Coordinate Descent with Coupled Constraints

Introduction For many big data applications, a relatively small parameter vector θ ∈ Rn is determined to fit a model to a very large dataset with N observations. We consider a different motivating problem in which both n and N are large. Thus, both batch optimization techniques and many stochastic techniques that require working with the entire θ vector (e.g. mirror descent methods) are too inefficient. Coordinate descent methods work only with a single coordinate of θ at each iteration. However, standard coordinate descent methods typically assume that the constraint sets for different coordinates are independent. We perform coordinate descent in the presence of coupled constraints. Specifically, we propose a simple algorithm that works with a group of k coordinates at each iteration; the minimum sufficient value of k depends on the constraint set. We prove convergence of the algorithm for polyhedral constraint sets and show experimental results over the the probability simplex.